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A  SIMULATION-BASED  TOOL  TO  TRAIN  RAPID  DECISION-MAKING 
SKILLS  FOR  THE  DIGITAL  BATTLEFIELD 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army’s  Future  Force  Warrior  program  exploits  opportunities  made  possible  by 
advances  in  our  capacity  to  quickly  gather,  organize,  and  distribute  battlespace 
information  available  from  multiple  sensor  and  database  systems.  As  this  transfer  of 
information  and  communication  technology  into  the  Future  Force  is  planned  and 
executed,  there  is  a  concurrent  need  to  develop  training  tools  that  will  enhance  the 
cognitive  skills  required  to  make  rapid  decisions  in  the  future  information-rich  operating 
environment.  This  report  documents  research  that  developed  a  computer-based 
simulation  tool,  called  the  Simulated  Field  Exercise  (SimFX)  tool,  to  train  small  unit 
leaders  to  resolve  ambiguous  or  contradictory  sensor  readings,  fuse  disparate  sources 
of  information,  and  employ  remote  sensors  -  whether  robotic  or  human  -  to  the  greatest 
effect  in  a  tactical  situation. 

Procedure: 

The  development  of  the  SimFX  software  was  guided  by  adherence  to  three  major 
themes  and  techniques.  SimFX  uses  outcome-driven  simulation  that  exploits  the 
cognitive  realism  that  results  from  engaging  students  in  a  story  or  vignette  in  which  they 
make  a  series  of  decisions  that  affect  how  the  story  plays  out.  The  development  of 
branching  storylines  give  the  students  access  to  multiple  sources  of  some  times 
potentially  conflicting  information  at  each  decision  point  and  advance  them  from  one 
decision  point  to  the  next.  Finally,  several  techniques  were  used  to  minimize  the 
combinatory  explosion  that  could  occur  keeping  track  of  the  multiple  paths  that  are 
possible  though  the  storyline. 

Findings: 

The  principle  product  of  this  research  was  the  SimFX  software  application  that  consists 
of  two  components.  The  Author  component  helps  training  developers  to  create  and 
modify  branching  storylines,  each  composed  of  linked  decision  nodes  that  form  the 
basis  for  training  scenarios  that  will  achieve  the  objectives  of  their  respective  training 
programs.  The  Player  component  presents  the  training  scenarios  to  the  student  and 
records  their  decisions  and  the  decision  outcomes.  The  research  also  produced 
separate  hard  copy  user  guides  and  tutorials  for  the  user  of  the  Author  and  the  Player 
components  of  SimFX.  The  usability  and  potential  effectiveness  of  both  SimFX 
components  have  received  favorable  reactions  from  participants  in  a  series  of  beta 
tests,  to  include  a  hands-on  workshop  conducted  for  a  broad  cross  section  of  trainers 
and  training  developers  at  Fort  Banning,  Georgia. 
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utilization  and  Dissemination  of  Findings: 

The  SimFX  tool  and  the  results  of  a  preliminary  training  evaluation  of  SimFX  have  been 
presented  to  senior  leaders  of  Infantry  training  and  training  development  at  Fort 
Banning,  Georgia.  The  use  of  existing  and  some  newly  created  SimFX  training 
scenarios  is  being  investigated  in  ongoing  research  at  Fort  Banning  for  training  Infantry 
squad  and  platoon  leaders.  Further,  a  series  of  hands-on  workshops  is  being 
conducted  across  Fort  Banning  and  elsewhere  to  ensure  that  trainers  and  training 
developers  are  aware  of  how  they  can  use  SimFX  to  contribute  to  their  respective 
training  objectives. 
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A  SIMULATION-BASED  TOOL  TO  TRAIN  RAPID  DECISION-MAKING 

FOR  THE  DIGITAL  BATTLEFIELD 


Introduction 

Background 

The  Infantry  Forces  Research  Unit  of  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  has  been  conducting  research  to  evaluate  the  training 
potential  of  desktop  simulations  of  dismounted  Infantry  operations  (Beal,  2005;  Beal  & 
Christ,  2004,  2005;  and  Centric,  Beal,  &  Christ,  2005).  The  desktop  simulations 
evaluated  were  developed  to  provide  Infantry  leaders  with  opportunities  to  experience 
realistically  the  consequences  of  executing  an  operations  order  and  the  challenges 
inherent  in  making  hasty  changes  to  those  orders  in  response  to  emerging  tactical 
conditions  in  the  current  or  contemporary  operating  environment  However,  in  keeping 
with  the  Army’s  modernization  plan,  there  is  a  need  to  develop  and  evaluate  desktop 
training  tools  that  can  enhance  the  types  of  cognitive  skills  required  to  make  rapid 
decisions  in  the  projected  future  operating  environment.  The  Army’s  Future  Force 
concept  exploits  the  enormous  opportunities  made  possible  by  advances  in  our  capacity 
to  quickly  gather,  organize,  and  distribute  battlespace  information. 

Several  years  ago  the  Infantry  Forces  Research  Unit  developed  a  topic 
statement  for  the  Small  Business  Innovation  Research  (SBIR)  program  that  asked  for 
the  development  of  a  computer-based  system  that  could  be  used  to  train  rapid  decision¬ 
making  skills  of  small  unit  leaders  regardless  of  the  level  of  technology  used  in  their 
operating  environment.  The  training  tool  was  to  be  initially  developed  for  use  by 
dismounted  Infantry  platoon  leaders  but  also  to  have  the  capability  to  be  used  by 
leaders  at  both  higher  and  lower  echelons.  The  objective  of  the  new  training  tool  was  to 
develop  the  leader’s  ability  to  access,  integrate,  and  effectively  use  information  from 
multiple  sources  to  improve  his  decision-making  proficiency.  Special  importance  was 
placed  on  the  role  of  the  information  provided  or  available  to  help  the  leader  make  the 
best  possible  decisions,  without  concern  for  the  format  and  structure  of  the  information 
(i.e.,  its  analog  or  digital  format)  or  the  means  by  which  the  information  was  presented 
(i.e.,  the  user-system  interface  or  “knobology”).  Finally,  the  training  tool  was  to  be 
capable  of  operating  on  any  Microsoft  Windows  operating  system  and  to  use  training 
scenarios  that  could  be  developed  by  training  developers  without  special  software 
development  skills.  Based  on  the  quality  of  the  background  work  and  plans 
accomplished  during  a  Phase  I  effort,  a  Phase  II  SBIR  contract  for  this  topic  was 
awarded  to  Micro  Analysis  and  Design,  Inc. 

Products  of  the  Phase  II  SBIR  Contract 

The  principal  product  of  this  SBIR  contract  meets  all  the  objectives  of  the  SBIR 
topic.  The  principal  product  is  a  software  application,  titled  Simulated  Field  Exercise 
(SimFX)  Tool.  In  addition  to  the  SimFX  software,  we  produced  a  user  guide  and  tutorial 
for  those  who  would  train  using  SimFX,  the  SimFX  Player  User  Guide  and  Tutorial,  and 
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a  companion  document  for  the  training  developer,  the  SimFX  Author  User  Guide  and 
Tutorial^  A  compact  disc  containing  the  SimFX  software  and  printed  copies  of  the  two 
user  guides  and  tutorials  (viz.,  Archer,  Brockett,  McDermott,  &  Warwick,  2006a,  2006b, 
2006c)  are  available  with  distribution  limited  to  U.S.  Government  Agencies  only  until 
March  31 , 2010.  These  research  products  can  be  obtained  from  the  Private  Scientific 
and  Technical  Information  Network  (STINET)  of  the  Defense  Technical  Information 
Center  (DTIC).  The  two  user  guides  and  tutorials  are  also  contained  in  the  SimFX  tool 
compact  disc. 


Purpose  of  This  Report 

In  keeping  with  the  requirements  of  the  SBIR  contract,  this  research  report  was 
developed  to  document  the  Phase  II  work  in  a  final  report  that  would  be  available  for 
unlimited  and  unrestricted  distribution  from  DTIC’s  public  STINET.  Consequently,  this 
report  describes  (a)  some  significant  questions  that  we  had  to  resolve  as  we  began  this 
research  and  development  effort,  (b)  the  rationale  and  approach  used  in  developing  the 
SimFX  tool,  (c)  the  SimFX  tool  itself,  and  (d)  the  results  of  efforts  we  undertook  to 
evaluate  the  usability  of  SimFX. 

Questions  Raised  by  the  Need  to  Train  Leaders  to  Use 
Technologies  Still  in  Development 

Since  the  transformation  in  technologies  required  to  support  the  digital  battlefield 
projected  for  the  Future  Force  is  not  yet  complete,  we  had  to  confront  two  significant 
questions  as  we  developed  methods  that  could  be  used  today  to  train  a  warrior  using 
tomorrow’s  technology.  First,  there  was  a  question  of  whether  the  transformation  in 
technology  would  lead  to  a  shift  in  the  nature  of  the  small  unit  leader’s  tactical  decision 
making.  The  very  fact  that  the  original  solicitation  called  for  the  development  of  new 
training  methodoiogies  suggested  that  the  problem  here  might  not  just  be  a  question  of 
familiarizing  the  warrior  with  new  pieces  of  digital  equipment  -  the  knobology  of  a 
system  -  but  rather  that  decision-making  processes  that  were  once  intuitive  might 
become  more  analytical  as  more  information  is  presented  to  the  small  unit  leader. 

Second,  having  originally  proposed  a  simulation-based  approach,  we  had  to 
address  the  question  of  how  much  realism  would  be  needed  in  the  simulation  to  ensure 
an  engaging  training  experience  in  which  the  appropriate  skills  would  be  acquired.  As 
computers  have  become  cheaper  and  more  powerful,  the  trend  in  simulation-based 
training  has  been  to  create  increasingly  immersive  “virtual  realities”  on  the  assumption 
that  a  highly  realistic  simulation  environment  will,  ipso  facto,  ensure  that  the  appropriate 
training  occurs  (i.e.,  the  student  will  not  simply  learn  how  to  “game”  the  simulation). 
While  an  immersive  flight  simulator  might  provide  the  student  an  opportunity  to  learn  the 
subtle  perceptual  and  motor  skills  needed  to  keep  an  airplane  in  the  air,  it  was  not  clear 
to  us  that  a  similar  approach  would  help  an  Infantry  leader  to  develop  the  cognitive  skills 
required  to  fuse  information  from  various  sources. 


'  In  this  report,  the  user  of  the  training  component  of  SimFX  is  calied  the  trainee,  student,  or  leader.  The 
user  of  the  authoring  component  of  SimFX  is  called  the  author,  trainer,  or  training  developer. 
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In  fact,  this  insight  helped  us  see  the  relationship  between  these  two  questions. 
More  specifically,  we  recognized  that  the  development  of  a  highly  realistic  simulation  of 
technologies  that  do  not  yet  exist  is  not  only  self-defeating,  but  also  unnecessary. 
Instead,  we  pursued  development  of  a  cognitively  engaging  simulation  environment,  in 
which  information  could  be  presented  rather  abstractly  (i.e.,  independently  of  the  user- 
system  interface  or  knobology  of  an  envisioned  system)  so  that  a  leader  would  be 
forced  to  make  decisions  under  specific  conditions.  This  approach  reflects  the  view  that 
repeated  experience  is  the  best  way  to  train  decision-making  skills  so  that  a  required 
cognitive  process  that  might  initially  be  analytic  and  labored  can  become  more  intuitive 
and  automatic.  Role  of  practice  and  feedback  on  the  development  of  expert  human 
performance  has  been  well  documented  by  Ericsson,  Krampe,  and  Tesch-Roemer 
(1993),  and  has  more  recently  been  extended  to  adaptive  thinking  behaviors  of  military 
commanders  (Shadrick  &  Lussier,  2004)  and  to  naturalistic,  intuitive  decision  making 
(Klein,  2003).  Moreover,  there  is  far  less  overhead  needed  to  engender  cognitive 
realism  in  a  simulation  (as  described  in  the  next  section)  than  is  required  for  immersive 
realism  in  a  virtual  simulation.  Development  of  an  immersive,  virtual  reality  simulation 
can  require  large  amounts  of  time  and  money,  as  well  as  skilled  software  engineers. 

The  overhead  incurred  in  developing  virtual  simulations  make  them  hard  to  maintain 
and  almost  impossible  to  adjust  for  changes  that  inevitably  occur  in  required  training 
tasks  and  conditions. 


Research  Approach 

We  describe  three  sets  of  interrelated  issues  and  techniques  in  this  research  and 
development  effort  in  detail  below  before  turning  to  a  more  general  description  of  tool 
itself.  These  three  aspects  of  the  research  approach  guided  our  development  of  the 
SimFX  tool. 


Cognitive  Realism 

Realism  is  an  essential  component  of  simulation-based  training.  For  many 
computer-based  simulations,  this  realism  is  accomplished  with  the  construction  of  a 
highly  detailed,  carefully  rendered,  synthetic  or  virtual  environment  coupled  with  some 
sort  of  input  device  that  allows  the  student  to  interact  with  the  simulated  environment. 
This  virtual  reality  permits  the  student  to  explore  the  simulated  training  environment  in 
real-time,  in  perceptual  and  response  situations  similar  to  those  that  may  be 
encountered  in  the  real  world.  The  actual  sequence  of  situations  encountered  in  the 
immersive  virtual  simulation  is  determined  by  propagating  the  effects  of  student 
decisions  and  actions  in  a  predictive  model  that  includes  autonomous  intelligent  support 
and  opposition  agents.  In  principle,  anything  the  student  might  do  in  the  actual 
environment  could  be  done  via  simulation  in  a  virtual  environment.  As  long  as  the 
simulated  environment  reflects  the  salient  interactions  of  the  actual  environment,  the 
student  can  gain  valuable  experience  performing  tasks  that  are  either  too  dangerous  or 
too  expensive  to  perform  in  the  actual  environment.  While  effective  for  some  types  of 
training,  immersion  in  a  virtual  reality  comes  with  its  own  issues  and  significant 
overhead  that  do  not  justify  its  application  in  every  training  domain.  (In  fact,  it  is  not 
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clear  that  highly  realistic  synthetic  environments  provide  useful  training  for  dismounted 
Infantry  leaders,  cf.,  Beal  &  Christ,  2004  and  Pleban  &  Salvetti,  2003). 

Outcome-driven  simulation  has  recently  emerged  as  one  possible  alternative  to 
an  immersive  simulation  (Gordon,  2004).  In  outcome-driven  simulation  the  goal  is  no 
longer  to  immerse  the  student  in  a  predictive,  model-driven  virtual  reality  but,  rather,  to 
exploit  the  cognitive  realism  that  follows  from  engaging  the  student  in  a  story  or 
vignette.  The  student  must  make  a  series  of  decisions  that  moves  the  story  forward  in 
time  to  new  situations  that  are  relevant  to  the  training  objectives.  The  user-defined 
movement  through  the  story  ultimately  affects  how  the  story  plays  out.  Outcome-driven 
simulation  trades  the  continuous  environment  of  virtual  reality  for  a  series  of  discrete 
choice  points  built  into  a  narrative  structure.  By  scripting  together  a  series  of  choice 
points  in  a  branching  storyline,  the  training  developer  maintains  control  over  the 
interactions  between  student  and  simulation.  The  branching  storyline  ensures  that  the 
student  will  encounter  specific  decisions  at  specific  times  rather  than  when  they  might 
be  required  during  unprescribed  movements  through  a  virtual  environment.  However, 
crafting  the  branching  storyline  that  constitutes  an  outcome-driven  simulation  places  a 
burden  on  the  developer  to  come  up  with  an  engaging  yet  tractable  scenario.  If  the 
training  developer  constructs  a  scenario  with  too  few  choice  points  he  runs  the  risk  of 
constructing  a  simulation  that  is  no  more  engaging  than  a  short  multiple  choice  exam. 

At  the  other  extreme,  if  the  developer  tries  to  string  together  too  many  choice  points  he 
will  quickly  find  himself  lost  in  a  combinatorial  explosion  of  branches.  The  training 
developer  must  strike  a  balance  between  engaging  the  student  and  managing  the 
complexity  of  a  scenario  while  maintaining  some  semblance  of  continuous  flow  and 
believability  throughout  the  scenario,  no  matter  which  choices  the  trainee  makes. 

Scenario-Based  Training 

Although  a  good  deal  has  been  written  recently  about  the  impacts  of  digital 
technologies  and  their  implications  for  training,  our  work  was  motivated  by  the  well 
established  principle  that  expertise  is  generally  built  on  a  foundation  of  practical 
experience.  So,  rather  than  focus  training  on  the  specifications  and  capabilities  of  new 
digital  technologies  -  the  knobology  of  new  technology  -  we  set  out  to  provide  students 
with  computer-based  scenarios  that  would  force  them  to  resolve  ambiguous  or 
contradictory  sensor  readings,  fuse  disparate  sources  of  information,  filter  information, 
manage  resources  (e.g.,  time,  network  bandwidth)  and  learn  how  to  employ  sensors  to 
the  greatest  effect  in  a  tactical  situation. 

For  example,  at  one  decision  point  we  might  ask  the  student  to  pick  among  three 
routes  to  a  waypoint.  The  paths  are  presented  on  an  electronic  display  of  a  map.  The 
student  has  the  ability  to  query  various  information  sources.  In  addition  to  traditional 
information  sources  (e.g.,  an  operations  order,  radio  communications,  map  overlays), 
the  student  can  query  unattended  acoustic  sensors,  visual  reconnaissance  from 
unmanned  air  and  ground  vehicles,  and  spot  reports  from  a  densely  connected 
communications  network.  Choosing  the  correct  path  means  querying  the  appropriate 
sensor  and  making  good  use  of  the  information  it  provides.  In  this  case,  the  situation 
was  crafted  so  that  the  student  must  recognize  that  the  indication  of  foot  traffic  reported 
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by  an  unattended  acoustic  sensor  in  the  vicinity  of  one  route  is  inherently  ambiguous 
and  that  the  determination  of  whether  it  is  due  to  enemy  or  friendly  activity  along  the 
route  depends  on  querying  another  sensor  -  perhaps  inspecting  recent  aerial 
reconnaissance.  The  choice  can  be  further  complicated  by  layering  tactical 
considerations  and  time  management  demands  (e.g.,  the  shortest  route  offers  less 
cover). 


Although  seemingly  straightforward,  implementing  this  decision  point  depended 
on  the  solutions  to  several  interrelated  questions.  First,  we  had  to  decide  how 
information  would  be  presented.  While  we  wanted  to  preserve  the  “look  and  feel”  of  the 
information  sources,  we  didn’t  want  the  student  to  become  mired  in  the  painstaking 
analysis  of  a  grainy  reconnaissance  photograph  or  the  interpretation  of  a  particular 
acoustic  signature  in  a  noisy  signal.  Instead,  we  opted  to  present  information  from 
these  sources  abstractly  (usually  as  text-based  reports  from  a  notional  intelligence 
analyst  who  reviews  sensor  data),  to  emphasize  how  the  student  should  integrate  such 
facts  once  presented  rather  than  train  interpretation  of  raw  data.  More  generally,  the 
abstract  representation  reflects  the  desire  to  steer  away  from  a  detailed  underlying 
model  where  a  consistent  “world  state”  can  be  maintained  and  presented  to  the  student 
(via  additional  and  comparatively  complex  sensor  models).  Instead,  the  training 
developer  simply  specifies  the  information  provided  to  the  student  at  each  decision 
point,  tweaks  the  simulated  world  as  necessary  (e.g.,  adding  enemies  at  a  location, 
removing  assets),  much  in  the  same  way  that  an  observer-controller  will  change  the 
course  of  a  live  training  exercise  to  suit  the  training  objectives.  But  while  the  training 
developer  gains  greater  control  of  the  simulation  in  this  way,  it  comes  at  a  cost.  Without 
detailed,  underlying  models  to  maintain  a  consistent  world  state,  it  falls  on  the  training 
developer  to  manage  the  complexity  and  consistency  of  the  unfolding  scenario.  As 
indicated  above,  managing  this  complexity  is  difficult. 

Managing  Complexity  in  Scenarios 

Even  with  only  a  few  choices  at  each  decision  point,  keeping  track  of  all  the 
possible  paths  through  a  scenario  may  become  unmanageable  after  a  handful  of 
decisions.  While  some  degree  of  combinatorial  explosion  is  inevitable,  it  can  be 
minimized  in  a  number  of  ways.  First,  as  Gordon  (2004)  describes,  a  branching 
scenario  can  be  pruned  by  introducing  “chapters”  whereby  a  series  of  decisions 
ultimately  funnel  back  to  a  single  decision.  For  example,  we  ask  the  student  a  short 
series  of  questions,  each  of  which  asks  where  he  would  move  to  next,  given  the 
available  information  sources  (which  can  change  from  decision  to  decision).  But  rather 
than  ramify  the  student’s  decisions  throughout  the  entire  scenario,  we  introduce  a  new 
series  of  questions  by  discontinuously  moving  the  student  to  a  new  location  that  could 
plausibly  be  reached  no  matter  which  route  the  student  chose  previously. 

A  second  technique  for  minimizing  combinatorial  explosion  is  simply  to  avoid  it  in 
the  first  place  by  posing  non-branching  decisions.  Such  decisions  either  ask  the 
student  to  provide  factual  responses  about  digital  technologies  (e.g.,  “Can  your 
unmanned  acoustic  sensor  field  at  the  objective  detect  truck  traffic  on  the  road  just  east 
of  the  objective?”)  or  to  estimate  the  resources  required  to  execute  particular  phases  of 
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the  mission.  Alternately,  we  could  use  rhetorical  strategies  to  force  the  student  to 
deepen  his  thought  about  the  tactical  situation  (e.g.,  “Have  you  considered  how  your 
operational  tempo  would  be  affected  if  you  were  flanked  en  route  to  the  objective?”). 
While  the  student  will  be  prompted  for  a  response,  the  response  does  not  change  how 
the  scenario  advances. 

Finally,  borrowing  techniques  from  the  gaming  community,  we  have  found  it  is 
possible  to  present  the  student  with  a  seemingly  genuine  decision  (i.e.,  a  choice  that 
affects  outcomes)  without  having  to  represent  those  outcomes  in  the  scenario.  The  trick 
here  is  not  to  predicate  the  outcome  of  the  decision  on  what  the  student  actually 
chooses,  but  rather,  on  what  the  student  knew  or  should  have  known  before  he  made 
his  choice  (which  we  can  infer  by  keeping  track  of  which  information  assets  were 
queried).  In  much  the  same  way  as  a  video  game  designer  will  program  a  monster  to 
appear  in  whatever  room  the  player  enters,  we  can  ensure  that  bad  things  will  happen 
whenever  a  student  fails  to  make  the  best  use  of  the  information  assets  at  his  disposal. 

Returning  to  our  earlier  example,  independent  of  the  route  the  student  actually 
chooses,  we  can  introduce  an  enemy  ambush  whenever  the  student  fails  to 
disambiguate  the  reports  from  his  acoustic  sensors.  Conversely,  the  enemy  will  be 
absent  whenever  the  appropriate  combination  of  sensors  is  queried  and  we  will  reward 
the  student  for  recognizing  the  original  ambiguity,  thus  reinforcing  the  training  objective. 
While  this  style  of  question  requires  the  training  developer  to  specify  training  feedback 
(i.e.,  outcomes)  for  a  potentially  large  number  of  sensor  combinations,  it  allows  large 
parts  of  scenarios  to  be  developed  without  any  branching,  and  so  the  level  of  effort 
tends  to  grow  linearly  rather  than  exponentially  in  the  depth  of  the  scenario. 


Description  of  the  SimFX  Tool 

The  development  of  SimFX  took  advantage  of  all  three  aspects  of  the  research 
approach.  SimFX  is  actually  two  software  components:  a  Player  component  that 
presents  a  training  scenario  to  a  student  and  an  Author  component  that  allows  the 
trainer  to  build  the  training  scenario  that  meets  the  training  objectives.  The  following 
sections  describe  both  the  Player  and  the  Author  components. 

From  the  Trainee’s  Perspective 

We  currently  envision  two  basic  types  of  decision  training  exercises.  However, 
SimFX  is  flexible  enough  that  we  would  not  be  surprised  if  training  developers  find  other 
ways  to  use  it  to  train  decision-making  skills. 

The  first  type  of  exercise  is  based  on  the  branching  storyline  that  we  have  been 
discussing  in  this  report.  In  this  type  of  exercise,  the  trainee  is  first  given  a  description 
of  the  mission.  This  description  can  range  from  a  brief  and  informal  statement  of  the 
purpose  of  the  mission  with  expectations  of  what  constitutes  a  successful  mission  to  a 
formal  five  paragraph  operations  order.  When  the  trainee  is  ready  to  begin  the  mission, 
the  scenario  jumps  to  the  first  decision  point.  Using  the  player  component  of  SimFX, 
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the  student  interacts  with  the  simulation  via  a  decision  dialog  window  (see  the  upper  left 
window  in  Figure  1 )  and  a  map  display  with  buttons  for  querying  sensors  (see  the  large 
window  on  the  right  side  of  the  figure).  The  decision  dialog  provides  a  description  of  the 
current  situation,  and  a  set  of  alternative  courses  of  action.  After  reading  the  decision 
prompt  and  information  contained  in  the  map  display,  the  student  may  query  one  or 
more  sensors  (human  or  robotic)  before  he  selects  an  alternative  and  presses  the 
Continue  button.  The  selected  sensors  may  provide  additional  information  in  a  text 
messages  window  (see  the  window  at  the  bottom  of  Figure  1)  that  can  help  the  student 
make  an  optimal  decision.  Based  on  the  student’s  inputs,  SimFX  displays  the  next 
decision  dialog  window.  This  dialog  window  first  presents  a  critique  of  the  student’s 
previous  decision  and  a  narrative  fragment  describing  the  consequences  of  that 
decision  in  terms  of  the  unfolding  story.  Secondly,  this  decision  dialog  window  moves 
the  scenario  to  the  next  decision  point  by  describing  the  new  situation  and  a  new  set  of 
decision  alternatives.  This  process  continues  until  the  mission  ends  in  success  or 
failure. 


Decision  Time  Remakvrw; 


TraveBng  through  e  heaviy  Forested  area,  you  come  «toss  the  burring 
remairts  of  what  looks  Hka  one  of  the  Humvees  from  your  Scout  section.  Wihat 
do  you  do? 


O  up  «  hasty  defense 

O  Swftch  to  bounding  overwatch  and  continue  mission 


Figure  1.  A  SimFX  screen  typical  of  those  presented  in  a  story-based  scenario.  In 
this  figure,  a  decision  dialog  window  is  shown  in  the  upper  left,  a  map  and  sensor 
query  window  is  shown  on  the  right,  and  a  text  message  window  is  shown  at  the 
bottom. 
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The  second  type  of  training  exercise  is  called  deliberate  practice.  Deliberate 
practice  exercises  do  not  follow  a  branching  storyline.  Trainees  are  presented 
repeatedly  with  the  same  stand-alone  decisions  but  with  different  information 
accompanying  each  decision.  The  trainee  still  needs  to  analyze  different  pieces  of 
information  to  make  a  good  decision.  For  example,  when  a  leader  is  given  visual 
information  about  a  situation  from  a  source  other  than  his  direct  perception  of  the 
situation,  such  as  a  photograph  or  a  segment  of  streaming  video,  it  is  very  easy  for  the 
leader  to  become  confused  and  disoriented  about  the  point  of  view  that  is  depicted. 
Deliberate  practice  exercises  could  show  the  trainee  aerial  photographs  of  the  same 
situation  taken  from  different  perspectives  and  ask  him  questions  regarding  the  different 
photographs.  Feedback,  along  with  an  explanation  of  the  relevant  cues,  gives  the 
trainee  practice  at  fusing  information  from  two  different  perceptual  points  of  view. 

Figure  2  shows  two  aerial  photograph  of  the  same  urban  scene  taken  from  different 
visual  perspectives.  The  decision  dialog  window  identifies  a  truck  shown  in  each 
photograph  and  asks  the  trainee  if  it  is  the  same  truck  seen  from  two  different  directions 
or  if  the  two  photographs  show  two  different  trucks.  Trainees  can  be  shown  many  sets 
of  photos  with  similar  questions  to  practice  making  visual  sense  of  information  from 
different  perspectives  (to  include  combinations  of  aerial  and  ground-level  views). 


Figure  2.  A  SimFX  screen  used  in  a  deliberate  practice  exercise.  A  decision  window 
is  shown  in  the  upper  left  and  a  high  altitude  aerial  photograph  in  the  upper  right.  Two 
low  level  aerial  photographs  are  shown  for  comparison  at  the  bottom  of  the  screen. 
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From  the  Training  Developer’s  Perspective 

Authoring  Process 

In  order  to  create  a  story-based  experience  for  the  student,  the  author  constructs 
a  branching  storyline  composed  of  linked  decision  nodes.  At  each  decision  node,  the 
scenario  author  must  confront  the  student  with  a  decision  to  make,  provide  some 
context  to  orient  the  student  in  the  story,  make  available  some  information  assets  to 
assist  in  making  the  decision,  and  provide  some  narrative  fragments  that  critique  the 
student’s  choices  and  describe  their  consequences.  The  SimFX  Author  User  Guide  and 
Tutorial  (Archer  et  al.,  2006c)  provides  extensive  guidance  on  the  important  points  to 
consider  at  each  decision  point.  In  addition,  an  authoring  template,  such  as  the  one  we 
created  for  our  sample  scenario  builders  (see  Appendix  A),  can  be  created  to  serve  as  a 
performance  aid  during  the  development  of  SimFX  scenarios. 

Typically,  the  author  begins  by  sketching  out  the  overall  shape  of  the  story. 
Gordon  (2004)  suggests  a  method  that  involves  distilling  teaching  points  from  a 
collection  of  anecdotes,  setting  up  decisions  to  support  those  teaching  points,  and  then 
crafting  the  story  around  the  decisions.  Our  process  was  similar  but  more  iterative 
since  we  repeatedly  found  ourselves  in  a  chicken  and  egg  situation,  wanting  to  build  the 
story  around  decision  points,  but  needing  a  basic  storyline  in  place  to  motivate 
decisions.  We  found  it  useful  to  start  with  what  we  called  the  “happy  path,”  in  which  the 
student  makes  only  good  choices  and  succeeds  in  his  mission.  Then,  we  would  go 
back  and  introduce  branches,  based  on  suboptimal  decisions,  which  lead  to  less 
favorable  alternative  endings.  Part  of  an  example  story  graph  for  one  of  our  scenarios 
is  shown  in  Figure  3.  This  figure  shows  a  decision  tree  with  eight  nodes,  most  of  which 
are  reached  through  decisions  made  by  the  trainee  but  some,  e.g.,  ambush,  are 


Figure  3.  An  example  of  an  eight-node  story  graph.  Successive 
nodes  connected  by  links  are  based  on  the  decision  options  selected 
by  the  trainee  or  a  combination  of  decisions  and  information  queries. 
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reached  though  a  combination  of  the  decision  options  the  trainee  selects  and  the 
specific  combination  of  information  assets  the  trainee  queries.  The  box  shown  around 
the  Choose  Route  decision  node  indicates  that  this  node  has  been  selected  by  the 
training  developer  for  authoring. 

The  authoring  of  an  individual  decision  node  begins  by  brainstorming  various 
ways  in  which  the  decision  prompt,  the  decision  alternatives  provided,  and  the 
information  sensors  made  available  can  be  combined  to  support  a  particular  teaching 
point  The  prompt  must  provide  some  “back-story”  to  make  the  decision  cognitively 
engaging,  and  the  decision  alternatives  must  all  seem  equally  plausible.  We  discovered 
that  decisions  having  several  answers  with  varying  degrees  of  "goodness"  made  for 
more  challenging  scenarios  than  ones  with  a  single  right  answer  and  several  wrong 
ones.  In  the  story  graph,  the  decision  alternatives  are  used  to  label  the  outbound  links 
from  a  decision  node.  The  scenario  author  specifies  the  properties  of  a  decision  node 
(e.g.,  its  name,  the  time  available  to  make  the  decision,  and  the  decision  prompt)  using 
the  Decision  Properties  dialog  box,  as  illustrated  in  Figure  4. 


r, 


^  Decision  Properties 


[Properti^l  Asset  Que^  Feedback 

Node  Label  Choose  Route 
Decision  Time  Limit  (sec)  1920 

Prompt  Using  your  map,  you  have  identified  three  possible  routes  to  the  bridge.  Which  route  ; 
seems  best?  "  } 


Question  Style  Multiple  Choice  Q  Essay 

Map  Overlay  Filename  InitiaIRoutes.gif 

1  Browse  |  ; 

Q  Clear  Text  Messages 

1  OK 

1  1  Cancel  | 

Figure  4.  A  decision  properties  dialog  box.  This  dialog  box  is 
keyed  to  the  Choose  Route  decision  node  shown  in  Figure  3. 


Referring  to  the  Choose  Route  decision  node  shown  in  Figure  3,  we  set  up  the 
decision  to  have  three  route  choices.  As  Figure  3  shows.  Routes  A  and  B  eventually 
lead  to  a  minefield,  while  Route  C  bypasses  it.  The  decision  prompt  we  used  for  this 
decision  node  is  shown  in  Figure  4.  It  provides  a  bit  of  context  but  no  direct  information 
about  any  of  the  routes.  The  student  must  not  only  choose  well  given  the  decision 
alternatives,  he  must  also  make  good  use  of  additional  information  available  to  him  from 
the  information  assets  provided.  In  this  specific  example  scenario,  we  provided  the 
decision  maker  with  an  Unmanned  Aerial  Vehicle  (UAV)  to  fly  over  any  or  all  of  the 
three  routes  and  an  Unattended  Ground  Sensor  (UGS)  located  near  Route  C. 
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The  teaching  point  we  wanted  to  support  in  this  decision  is  that  often  one  source 
of  information  is  not  enough  to  provide  a  clear  picture  of  the  situation.  However,  simply 
querying  all  available  information  assets  at  each  decision  point  is  not  a  viable  strategy 
either,  since  some  assets  -  the  UAV  for  example  -  can  be  costly  in  mission  time  and 
resources  to  deploy.  Therefore,  the  student  must  learn  to  discern  which  information 
sources  are  most  appropriate  to  query  for  a  given  situation. 

In  our  example,  the  student  must  choose  one  of  the  three  alternative  routes,  and 
can  decide  to  query  or  not  query  each  of  four  different  sources  of  information  -  aerial 
reconnaissance,  using  the  UAV,  on  each  of  the  three  routes,  plus  the  unattended 
ground  sensor.  SimFX  uses  the  Asset  Query  Feedback  tab  on  the  Decision  Properties 
dialog  box  shown  in  Figure  4  to  tabulate  compactly  the  various  combinations  of  decision 
choices  and  assets  selected,  and  to  specify  a  unique  result  for  each  combination  of 
student  inputs.  A  sample  table  of  the  Asset  Query  Feedback  dialog  box  for  the  Choose 
Route  decision  node  is  shown  in  Figure  5. 


Rules  Choice 

UGSl 

UAV/Route  B 

UAV/Route  A 

;  UAV/Route  C 

i  Result 

NextNode 

A 

Don't  Care 

Don't  Care 

No 

Don't  Care 

R1 

A 

Don't  Care 

Don't  Care 

Yes 

Don't  Care 

R2 

B 

Don't  Care 

No 

Don't  Care 

Don't  Care 

R3 

B 

Don't  Care 

Yes 

Don't  Care 

Don't  Care 

R4 

C 

No 

Don't  Care 

Don't  Care 

No 

R5 

Ambush 

C 

No 

Don't  Care 

Dont  Care 

Yes 

R6 

c 

Yes 

Don't  Care 

Don't  Care 

No 

R7 

c 

Yes 

Don't  Care 

Don't  Care 

Yes 

R8 

Figure  5.  A  decision  rules  table.  This  decision  rules  table  is  keyed 
to  the  Choose  Route  decision  node  shown  in  Figure  3. 

The  first  5  columns  in  the  rules  table  correspond  to  student  inputs  -  the  decision 
alternative  he  selects  and  the  sensors  he  does  or  does  not  query.  When  the  student 
makes  his  decision,  SimFX  examines  the  rows  in  the  table  in  order,  starting  at  the  top, 
looking  for  a  row  in  which  entries  in  the  first  5  columns  match  what  the  student  actually 
did.  In  addition  to  Yes  and  No  as  possible  values  for  a  sensor  query,  “Don’t  Care”  can 
be  used  to  indicate  that  it  is  irrelevant  whether  the  student  queried  the  corresponding 
sensor.  The  use  of  Don’t  Care  entries  can  greatly  reduce  the  number  of  rows  in  the 
decision  table,  thereby  reducing  its  complexity  and  the  effort  required  to  create  it. 

When  a  match  is  found,  a  corresponding  results  table  in  the  Assets  Query 
Feedback  dialog  box,  such  as  that  illustrated  in  Figure  6,  is  used  to  identify  the  text  the 
author  has  prepared  for  each  possible  result.  This  text  will  be  presented  to  the  student 
as  feedback  (i.e.,  the  critique  and  consequences  of  a  decision)  to  the  student  for  his 
inputs  for  this  decision  node,  along  with  the  prompt  for  the  next  decision  node.  Figure  6 
shows  some  examples  of  feedback  the  student  would  get  depending  upon  his  input  to 
the  Choose  Route  decision  node  of  the  story  graph  shown  in  Figure  3.  For  example,  if 
the  student  queries  the  UGS  but  not  the  UAV  along  Route  C,  then  chooses  Route  C, 
the  seventh  row  in  the  input  tabulation  table  matches  and  the  narrative  associated  with 
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Result  R7  (see  Figure  6)  is  displayed  as  feedback  to  the  student  for  his  inputs  to  the 
Choose  Route  decision  along  with  the  prompt  for  the  next  decision. 


Results  ID  Text 

The  map  showed  this  route  to  be  the  longest.  Moreover,  had  you  used  your  sensors,  you  would  have  learned 
of  the  presence  of  an  enemy  patrol  along  the  route.  En  route,  you  are  attacked  and  overwhelmed.  There  are 
R5  no  survivors, 


As  your  aerial  recon  indicated,  the  route  is  clear  except  for  some  local  noncombatants. 


As  the  report  from  your  unattended  ground  sensor  indicated,  there  has  been  recent  foot  traffic  along  the  route  ; 
.  It  would  have  been  wise  to  employ  your  UAV  to  attempt  to  determine  if  that  foot  traffic  represented  a  threat, 
R7  Fortunately,  you  arrive  at  the  bridge  without  incident. 


R8 

■<  : 


Good  choice.  Route  C  is  a  reasonable  route,  and  your  use  of  the  UAV  to  confirm  that  the  foot  traffic  detected 
by  the  UGS  was  not  a  threat  was  wise. 


>! 


Figure  6.  A  sample  of  a  decision  results  table.  This  decision  results  table 
is  keyed  to  the  Choose  Route  decision  node  shown  in  Figure  3. 

All  of  these  narrative  elements  -  the  prompt,  the  choices,  the  available  sensors, 
and  the  outcome  and  critical  feedback  -  must  support  the  particular  teaching  point 
associated  with  the  decision.  In  addition,  the  outcome  and  feedback  must  fit  seamlessly 
together  with  the  prompt  for  the  next  decision  in  the  graph,  so  that  the  student  is 
presented  with  a  coherent,  cohesive  story  from  beginning  to  end,  no  matter  which  path 
he  takes  through  the  story. 

We  discovered  that  constructing  a  scenario  which  is  not  easy  to  “game”  requires 
great  care.  As  mentioned  previously,  decisions  must  be  framed  in  a  way  that  the  best 
answer  is  not  obvious,  and  that  “always  query  all  sensors”  is  not  a  successful  strategy. 
Time  pressure  also  makes  the  simulation  hard  to  game  and  increases  student 
engagement.  In  SimFX,  we  created  two  countdown  clocks  -  one  that  limits  the  amount 
of  time  available  for  each  decision,  and  another  that  limits  the  total  time  available  to 
accomplish  the  mission.  Our  experience  with  an  early  beta  test  confirmed  that  time 
pressure  was  a  significant  factor  in  making  the  scenarios  hard  enough  to  be  interesting. 

While  we  have  not  introduced  the  concept  of  scoring  decision  performance  into 
this  version  of  SimFX,  it  would  be  straightforward  to  allow  the  author  to  attach  a  positive 
or  negative  value  to  each  row  in  decision  tables  such  as  that  illustrated  in  Figure  5,  and 
display  the  cumulative  score  at  the  end  of  a  training  session.  SimFX  does  create  a 
report  showing  the  decision  choices  and  information  asset  selections  of  each  student  at 
each  choice  point.  This  information  can  be  used  as  part  of  an  after-action  review  with 
an  instructor. 


SimFX  Authoring  Principles 

In  the  process  of  creating  multiple  training  exercises  we  have  learned  a  lot  about 
how  to  use  the  SimFX  tool  to  create  and  modify  story-based  and  deliberate  practice 
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scenarios.  Based  on  these  experiences,  we  have  developed  some  principles  that 
appear  to  be  quite  useful.  These  authoring  principles  are  described  in  subsequent 
paragraphs. 

Use  the  “back-story”  (decision  prompt)  to  provide  cues  and  motivate  the 
decision.  Back-story  describes  the  situation  which  calls  for  a  decision  and  perhaps 
provides  some  relevant  cues.  Digital  information  will  be  most  useful  to  small  unit 
leaders  when  they  are  in  the  middle  of  a  “planning  break”,  rather  than  actively 
navigating  terrain  or  engaging  the  enemy.  Therefore,  the  decision  prompt  should  cast 
the  trainee  in  the  middle  of  one  of  these  brief  pauses  in  mission  execution.  For 
example,  consider  "traveling  through  a  heavily  wooded  area,  you  come  across  the 
burning  remains  of  a  Humvee".  This  is  a  situation  that  calls  for  some  kind  of  action;  the 
trainee  should  not  merely  proceed  with  the  mission.  The  fact  that  the  area  is  heavily 
wooded  may  be  significant  (for  example,  using  a  UAV  may  be  ill-advised  because  of  a 
dense  tree  canopy). 

Craft  decision  options  carefully  to  support  the  decision’s  teaching  point 

Make  decision  alternatives  equally  plausible,  so  that  the  best  answer  is  not  obvious.  If 
the  trainee  can  guess  the  right  answer,  no  real  training  transfer  is  likely  to  occur.  Often, 
it’s  best  to  create  decision  options  in  “shades  of  gray”  -  optimal,  near-optimal,  and 
suboptimal  -  rather  than  one  “right”  and  several  ‘Swrong”  options.  This  approach  reflects 
the  truth  that  in  real  life,  there  is  almost  always  more  than  one  way  to  solve  a  problem. 
For  scenarios  to  be  realistic,  different  optimal  or  near-optimal  choices  should  have 
different  consequences  -  sometimes  ones  that  show  up  much  later  in  the  scenario.  Be 
careful  not  to  give  away  information  in  the  decision  options  that  the  trainee  should  be 
able  to  find  out  only  by  using  information  assets  -  for  example,  an  option  like  “dispatch 
SUGV  [small  unmanned  ground  vehicle]  to  observe  gathering  crowd”  when  the  trainee 
has  no  idea  there  is  a  crowd. 

Don't  mix  asset  queries  with  decision  options.  It  may  be  tempting  to  create 
decision  options  like  "Deploy  your  UAV"  and  "Send  out  a  Recon  Squad".  This  approach 
is  not  necessarily  incorrect,  but  it  fails  to  take  advantage  of  many  of  the  capabilities 
SimFX  has  to  create  engaging  scenarios.  If  asset  deployment  is  tied  to  decision 
options,  the  trainee  will  not  have  the  option  of  querying  none,  one,  or  both.  He  will 
always  have  to  pick  just  one.  Thus,  no  information  fusion  (combining  information  from 
multiple  assets)  can  be  taught.  Generally  speaking,  making  assets  available  to  the 
trainee  via  the  asset  toolbar,  rather  than  through  decision  options,  will  result  in  more 
interesting  scenarios. 

Choose  assets  so  that  the  trainee  must  fuse  the  right  information  in  the 
right  way.  In  the  future  battlespace,  leaders  will  need  to  reconcile  disparate  data 
sources,  filter  out  and  select  relevant  information,  diagnose  inconsistencies  and 
contradictions  across  the  data,  and  deal  with  ambiguous,  missing,  or  erroneous  sensor 
information.  Challenging  the  trainee  with  these  kinds  of  problems  will  result  in  engaging 
scenarios  that  result  in  training  transfer. 
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Set  up  situations  that  require  the  development  of  spatial  orientation  skills. 

Spatial  orientation  will  become  a  key  issue  and  skill  in  the  electronic  battlefield.  A 
leader  in  the  future  high  technology  force  will  need  to  understand  the  data  being  sent  to 
him  from  the  sensor’s  point  of  view  and  then  translate  that  into  information  that  he  can 
use  to  make  a  correct  decision.  If  the  leader  is  even  slightly  disoriented  as  to  where  a 
sensor  is  looking  with  relation  to  him,  the  results  could  be  disastrous. 

Use  time  pressure  to  create  a  sense  of  urgency  and  prevent  “gaming  the 
system.”  Adjust  the  time  available  for  each  decision  such  that  the  trainee  only  has  time 
to  query  the  particular  combination  of  information  assets  whose  output  you  want  the 
trainee  to  fuse  to  come  to  a  correct  decision.  This  prevents  the  trainee  from  using  the 
obvious  success  strategy  of  querying  every  asset  at  every  decision  point,  which  will 
simply  not  be  possible  in  a  real  combat  situation.  Querying  an  asset  should  carry  with  it 
a  cost  in  terms  of  time,  and  the  decision  should  usually  not  allow  enough  time  to  query 
all  information  assets.  In  addition,  identify  just  a  few  optimal  and  near-optimal  paths 
through  the  scenario,  and  adjust  the  between-decision  (link)  times  and  overall  mission 
time  available  so  that  it  is  just  barely  possible  to  complete  these  paths.  Adjust  the  link 
times  on  sub-optimal  paths  so  that  the  trainee  will  run  out  of  mission  time  if  he  takes 
them.  Using  time  pressure  in  this  way  will  make  your  scenarios  more  engaging  and 
more  likely  to  force  the  trainee  to  think  carefully  about  his  actions  and  learn  from  his 
mistakes. 


Evaluation  of  SimFX  Usability 

We  conducted  a  series  of  beta  tests  to  gauge  the  usability  of  SimFX  and  the 
likelihood  that  it  could  deliver  useful  training.  The  beta  test  procedures  and  findings  for 
both  the  SimFX  Player  and  the  SimFX  Author  are  described  below. 

Internal  Beta  Test  and  Focus  Group  for  the  SimFX  Player 

An  internal  beta  test  of  the  SimFX  Player  component  of  SimFX  was  conducted 
using  employees  of  Micro  Analysis  and  Design,  Inc.  The  goals  of  these  tests  were  to: 
(a)  Discover  confusing  aspects  of  the  Player  component;  (b)  Uncover  misconceptions 
about  Player  and  the  training  objectives;  and  (c)  Discover  inconsistencies  in  the 
scenario.  Six  personnel  participated  in  the  beta  test.  The  participants  varied  in  their 
level  of  computer  skills  and  their  familiarity  with  the  Army’s  Future  Force  systems.  The 
participants  included:  two  former  platoon  leaders  in  the  Army,  an  office  manager;  an 
entry-level  systems  engineer;  a  high-level  applications  engineer  who  works  on  military 
simulation  products;  and  a  senior  analyst  who  works  on  the  development  of  Future 
Force  robotic  systems. 

Participants  were  given  a  brief  description  of  the  purpose  of  the  SimFX  Player 
component  and  how  it  worked.  After  installing  the  software,  the  participants  went 
through  the  beta  test  scenario  multiple  times,  noting  anything  that  was  confusing  or 
unexpected.  This  activity  took  between  one  and  two  hours.  When  all  the  participants 
were  finished,  a  group  debrief  was  conducted  in  which  we  elicited  details  about  the 
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experiences  they  had  while  using  the  Player  component.  The  following  questions  were 
asked  at  the  debrief: 


•  In  your  words,  what  was  the  purpose  of  this  component  of  SimFX?  What 
would  you  guess  we  were  trying  to  accomplish? 

•  What  did  you  like  best  about  Player? 

•  Tell  about  a  time  when  the  things  did  not  work  how  you  expected?  Explain. 

•  Tell  about  a  time  when  you  where  confused?  Explain. 

•  What  was  frustrating  about  the  Player  component  of  SimFX  or  the  scenario? 

.  Was  there  a  time  when  something  happened  in  the  scenario  that  just  didn’t 

make  sense? 

•  If  you  could  change  one  thing  about  this  component  of  SimFX  what  would  it 
be? 

•  Tell  me  some  things  you  learned  about  using  robotic  sensors? 

•  What  do  you  know  now  that  you  didn’t  know  before  going  through  the 
scenario? 

•  We  want  to  make  it  easy  for  people  to  navigate  though  Player  and  understand 
what  actions  they  can  take.  What  advice  or  directions  should  we  give  people 
before  they  sit  down  to  use  Player? 

Participants  indicated  that  they  correctly  understood  that  the  purpose  of  the 
Player  component  of  SimFX  was  to  prepare  the  decision  maker  for  a  real  life  situation  in 
which  they  would  need  to  use  robotic  assets.  SimFX  helped  them  understand  what 
resources  they  had  and  how  to  use  them.  The  participants  also  learned  specific  things 
about  the  employment  of  assets.  These  would  vary  depending  on  the  scenario  and  the 
learning  objectives  of  the  developer.  As  an  example,  one  participant  said  they  learned 
to  look  for  text  messages  because  they  often  contained  critical  information.  Another 
participant  learned  that  UAVs  take  longer  to  gather  information  than  some  of  the  other 
resources. 

There  were  several  aspects  of  the  Player  and  scenario  that  were  confusing.  We 
modified  the  software  to  fix  confusions  related  to  the  scenario.  For  example,  an  asset 
may  have  been  available  at  one  decision  point,  not  available  at  the  second,  and  then 
available  again  on  third  decision.  We  revised  the  scenario  so  this  was  more  consistent 
or  at  least  explained  (i.e.,  the  unmanned  ground  vehicle  (UGV)  is  not  available  because 
of  bandwidth  limitations  in  hilly  terrain).  In  terms  of  the  Player  component,  participants 
agreed  that  they  needed  to  play  the  scenario  three  to  four  times  before  they  felt  like  they 
knew  what  they  were  doing.  In  addition,  there  were  multiple  assets  (such  as  text 
messages,  SITREPs,  and  biosensors)  that  the  participants  did  not  notice  the  first  time 
using  the  Player.  We  considered  this  to  be  beneficial  for  several  reasons.  First  of  all,  it 
is  our  not  our  intent  to  train  users  completely  in  one  run  on  a  scenario.  The  scenarios 
are  meant  to  be  exploratory  in  that  trainees  learn  by  trial  and  error  and  by  noticing 
trends  in  how  the  use  of  assets  and  information  impacts  outcomes.  Secondly,  we  know 
that  no  matter  how  well  designed,  the  systems  that  convey  digital  information  in  the 
future  will  have  features  that  are  confusing  and  non-intuitive  on  the  first  try.  By 
practicing  with  SimFX,  small  unit  leaders  will  be  better  prepared  to  deal  with  the 
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confusion  and  uncertainty  that  are  inherent  during  a  mission.  Small  unit  leaders  will 
learn  how  to  seek  the  information  they  need  and  which  types  of  information  are  most 
useful  in  different  situations. 

One  of  the  key  findings  from  these  in-house  beta  tests  was  the  fact  that  there 
was  not  enough  time  pressure.  In  almost  every  decision,  participants  had  plenty  of  time 
to  use  every  robotic  resource  available.  It  was  our  intent  that  participants  would  run  out 
of  time  if  they  queried  every  asset.  Therefore,  we  revised  the  timing  in  the  beta  (and 
subsequent)  scenarios  to  ensure  there  was  sufficient  time  pressure.  Participants 
identified  specific  technical  problems  such  as  the  inability  to  see  gridline  references  in 
the  initial  map.  They  also  made  suggestions  on  how  to  improve  the  interface,  such  as 
being  able  to  mouse  over  an  asset  to  get  additional  details  on  its  capabilities.  These 
problems  and  suggestions  were  addressed  in  subsequent  versions  of  SimFX. 

Overall,  the  internal  beta  group  agreed  that  SimFX  had  a  solid  approach.  The 
branching  seemed  robust  and  there  were  enough  paths  and  options  to  keep  the 
participants  engaged  for  multiple  runs  through  the  scenario.  Most  problems  were  fixed 
by  decreasing  the  decision  times  and  thereby  increasing  the  time  pressure. 

Beta  Test  for  the  SimFX  Author 

The  beta  test  of  the  SimFX  Author  component  was  conducted  over  a  six  month 
period  as  part  of  a  spiral  development.  Three  personnel  participated  in  a  beta  test  by 
using  the  SimFX  Author  to  create  a  scenario  from  scratch.  These  included:  a  subject 
matter  expert  contractor  with  20  years  active  duty  in  the  Army,  a  contractor  for  ARI,  and 
a  documentation  specialist.  Two  of  these  beta  testers,  the  Army  subject  matter  expert 
and  the  ARI  contractor,  used  SimFX  to  author  two  scenarios.  An  analyst  at  Micro 
Analysis  and  Design  served  as  the  fourth  beta  test  participant.  Instead  of  creating  a 
scenario  from  scratch,  this  participant  was  responsible  for  inputting  a  pre-defined 
scenario  into  SimFX.  The  scenario  had  been  previously  specified  using  the  Decision 
Template  we  developed  (see  Appendix  A)  and  needed  to  be  converted  into  a  SimFX 
scenario.  This  participant  also  used  SimFX  Author  to  make  adjustments  to  several  other 
scenarios  -  both  branching  stories  and  deliberate  practice  scenarios. 

The  beta  test  of  the  Author  component  occurred  at  different  times  in  its 
development.  We  implemented  most  of  the  suggestions  from  the  beta  testers.  Some 
suggestions  pertained  to  how  things  worked  in  SimFX.  For  example,  in  addition  to 
being  able  to  send  a  SITREP  to  higher  headquarters,  a  player  in  the  game  can  now 
send  an  order  to  subordinates.  Also,  training  developers  can  now  play  the  scenario 
they  are  creating  directly  in  the  Author  component  of  SimFX.  This  ability  to  verify  that 
the  scenario  is  working  as  intended  has  greatly  streamlined  the  scenario  development 
process.  Feedback  from  the  beta  testers  was  also  used  to  fix  minor  human  factors 
issues.  For  example,  the  appearance  of  buttons  was  revised  to  make  it  more  obvious 
that  they  were  buttons  and  labels  were  changed  to  make  their  functionality  more 
obvious. 
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Our  own  experience  using  the  author  component  of  SimFX,  coupled  with 
feedback  from  the  beta  group,  prompted  us  to  create  aids  to  assist  in  the  authoring 
process.  It  became  clear  that  authoring  scenarios  was  a  difficult  task,  whether  it  was 
done  within  SimFX  or  via  pencil  and  paper.  To  compensate  for  this,  we  created  a 
Decision  Template  (see  Appendix  A)  that  guides  a  scenario  developer  through  the 
important  considerations  of  every  decision.  In  addition,  a  SimFX  author  user  guide  was 
developed  that  walks  training  developers  through  the  authoring  process.  The  guide 
includes  a  tutorial  using  a  concrete  example,  as  well  as  authoring  principles  or  “lessons 
learned”  to  help  the  developer  get  the  most  impact  out  of  his  or  her  training. 

Beta  Test  for  SimFX  Player  and  Author  During  a  Workshop  at  Fort  Banning 

We  conducted  a  workshop  at  Fort  Banning,  Georgia,  with  a  group  of  30 
participants  drawn  from  a  broad  cross  section  of  the  Infantry  training  and  training 
development  communities.  While  a  detailed  description  of  the  Fort  Banning  beta  test  of 
SimFX  has  already  been  published  (Christ,  2006),  this  section  summarizes  the 
procedures  and  results  of  that  evaluation  so  that  all  the  relevant  usability  research  is 
reported  together  in  one  report. 

During  the  workshop  we  first  described  our  motivations  and  visions  for  SimFX. 
We  then  walked  the  participants  through  the  SimFX  beta  software  to  demonstrate  both 
the  training  and  authoring  capabilities  of  the  tool.  Workshop  participants  were  given  a 
chance  to  experience  a  variety  of  SimFX  training  scenarios  first  hand.  These  included  a 
story-based  scenario  aimed  at  training  general  information  fusion  skills,  and  two 
deliberate  practice  exercises  targeting  specific  skills.  The  participants  also  had  a 
hands-n  demonstration  of  how  the  Author  component  of  SimFX  could  be  used  to  create 
a  simple  three-node  scenario.  After  a  period  of  relatively  unstructured  exploration,  we 
asked  the  participants  to  complete  a  questionnaire. 

The  questionnaire  was  developed  to  capture  the  opinions  of  the  participants 
about  their  experiences  with  SimFX  during  the  workshop.  Successive  parts  of  the 
questionnaire  asked  the  respondents  to  rate  the; 

•  The  training  value  of  SimFX. 

•  The  extent  to  which  users  of  SimFX  would  be  personally  involved  with  the 
training. 

•  The  ease  of  use  or  the  usability  of  SimFX  for  training  and  for  editing  or 
authoring  a  training  scenario. 

Participants  were  also  asked  to  provide  written  comments  about  SimFX  in  terms  of  its 
advantages  and  disadvantages  as  an  aid  for  training  and  as  a  method  for  editing  or 
authoring  training  scenarios. 

Nineteen  participants  completed  the  questionnaire.  In  general,  the  opinions  they 
expressed  about  SimFX  were  positive.  Eleven  questions  asked  about  the  training  value 
of  SimFX  using  a  seven-point  scale  ranging  from  three  decreasingly  negative  opinions. 
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though  a  neutral  rating  category,  to  three  increasingly  positive  opinions.  Between  68 
and  95  percent  of  the  respondents  used  one  of  the  three  highest  rating  categories  to 
indicate  they  had  positive  opinions  about  the  training  value  of  SimFX.  Five  questions 
using  a  similar  seven-point  scale  asked  about  the  capability  of  SimFX  to  fully  engage  or 
involve  the  user.  Between  52  and  95  percent  of  the  respondents  used  one  of  the  three 
highest  rating  categories  for  these  items  to  indicate  that  they  believed  SimFX  would 
capture  the  attention  and  motivation  of  trainees.  Finally,  participants  were  asked  to 
indicate  their  level  of  agreement  with  twenty-five  positive  statements  about  the  usability 
of  SimFX  using  a  five-point  rating  scale.  Between  61  and  100  percent  of  the 
respondents  either  agreed  or  strongly  agreed  with  these  positive  statements  indicating 
that  they  believed  SimFX  was  usable  for  training  and  for  authoring  training  scenarios. 

The  respondents’  written  comments  about  SimFX  training  followed  a  pattern 
similar  to  their  rated  opinions,  with  a  majority  of  positive  comments  reflecting,  among 
other  things,  the  ease  of  use,  the  emphasis  on  the  training  decision-making  skills,  and 
the  potential  to  modify  existing  SimFX  scenarios  to  fit  new  training  requirements.  Some 
of  the  negative  comments  concerned  details  specific  to  the  training  scenario  we 
demonstrated.  In  particular,  many  of  the  respondents  were  unfamiliar  with  the 
capabilities  of  the  remote  and  robotic  sensor  technologies,  which  played  a  central  role  in 
our  demonstration  scenarios.  Other  negative  comments  hit  on  issues  common  to  many 
training  tools  such  as  the  difficulty  inherent  in  developing  good  training,  the  lack  of 
access  to  computers  in  the  field,  and  a  preference  for  live  training. 

Written  comments  about  the  authoring  capability  in  SimFX  were  less  positive, 
with  many  respondents  expressing  concerns  about  the  perceived  difficulty  of  authoring. 
However,  compared  to  the  discussion  of  training  with  SimFX,  the  authoring  discussion 
received  short  shrift  during  the  workshop,  so  these  comments  are  not  entirely 
surprising.  At  the  same  time,  some  respondents  indicated  that,  with  a  little  practice, 
they  thought  authoring  might  not  be  too  onerous,  especially  given  an  existing  scenario 
to  modify. 

Finally,  our  beta  test  at  Fort  Benning  uncovered  an  application  for  SimFX  we  had 
not  considered.  Several  participants  pointed  out  the  potential  for  using  SimFX  as  a 
general  method  for  creating  and  modifying  training  scenarios  that  could  be  used  in 
another  training  environment.  Quite  often,  such  development  is  undertaken  using  paper 
and  pencil  story  boards  or  carefully  constructed  PowerPoint  presentations.  The 
decision  point  editor  and  the  compact  representation  of  the  branching  logic  within  the 
authoring  component  of  SimFX  provides  a  more  flexible  scenario  development 
environment,  while  the  Player  component  of  the  SimFX  tool  could  serve  as  a  “playback” 
environment. 


Conclusion 

Given  the  findings  discussed  above  regarding  the  numerous  and  varied  beta 
tests,  as  well  as  the  experiences  that  our  project  team  have  had  throughout  the 
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development  cycle,  we  conclude  that  the  SimFX  tool  and  the  training  development 
methodology  surrounding  it  can  have  a  potentially  positive  impact  for  training  present 
and  future  small  unit  leaders  to  make  rapid  decisions  while  fusing  information  from 
disparate  sources.  The  SimFX  tool  puts  the  focus  on  cognitive  realism  instead  of 
immersive  virtual  realism  for  training,  resulting  in  a  light-weight  application. 

While  our  approach  places  a  non-trivlal  demand  on  the  training  developer  to 
produce  a  well-crafted,  outcome-driven  scenario,  the  required  effort  pales  in  comparison 
to  that  required  to  develop,  maintain,  and  use  more  immersive  simulation  environments. 
At  the  other  extreme,  we  are  inspired  by  a  generation  of  paper  and  pencil  exercises, 
called  Tactical  Decision  Games,  which  the  U.S.  Marines  use  to  get  students  to  suspend 
disbelief  and  engage  the  story  behind  the  training  scenario.  We  see  the  outcome-driven 
simulation  approach  that  we  have  built  into  the  SimFX  tool  as  a  middle  road  between 
highly  immersive  virtual  simulation  exercises  and  pencil-and-paper  exercises. 

Features  of  the  SimFX  authoring  component  also  emphasize  sound  educational 
principles  and  techniques  for  training  leaders  to  use  digital  technologies  (Graham  & 
Dyer,  2002),  such  as  the  use  of  well  crafted  teaching  points,  advanced  organizers  and 
constructive  feedback.  The  SimFX  approach  is  designed  to  be  used  directly  by  training 
developers  with  domain  expertise,  as  opposed  to  having  training  application 
development  primarily  in  the  hands  of  software  engineers. 

We  have  encountered  several  instances  where  people  were  interested  in  SimFX 
as  a  method  for  authoring  branching  scenarios  that  are  not  necessarily  for  training 
digital  information  fusion.  As  noted  earlier,  the  most  popular  means  for  general 
scenario  authoring  seem  to  be  paper  and  pencil  story  boards  or  PowerPoint 
presentations.  SimFX  provides  a  much  more  flexible  and  sophisticated  scenario 
development  and  playback  environment  than  is  possible  with  these  more  common 
authoring  methods. 

Finally,  we  conclude  that,  although  SimFX  was  developed  for  the  Army,  with 
Army  Infantry  leaders  in  mind,  it  could  also  easily  be  adapted  to  other  instances  where 
crucial  decisions  have  to  be  made  using  widely  varying  sources  of  information. 

Intended  users  of  SimFX  could  be  expanded  to  include  other  military  services,  police 
forces,  disaster  management,  and  homeland  security  incidents. 
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Appendix  A 
Decision  Template 


We  developed  the  decision  template  presented  in  this  appendix  to  serve  as  a 
performance  aid  for  individuals  who  were  creating  training  scenarios  using  the  Author 
component  of  the  Simulated  Field  Exercise  (SimFX).  The  decision  template  guides  the 
scenario  developer  through  the  important  considerations  that  need  to  be  made  at  every 
node  in  the  decision  graph. 
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Decision  Point  #n:  [Brief  title  describing  the  decision  to  he  made] 


Teaching  point: 

[General  principle,  teaching  point,  learning  objective] 

Application  of  teaching  point  in  this  decision: 

[How  do  the  specifics  of  the  way  this  decision  is  set  up  bring  out  the  teaching  point  or 
accomplish  the  learning  objective?] 

Decision  prompt: 

[The  actual  prompt  the  trainee  will  see  during  scenario  playback.  Usually  consists  of 
two  parts  -first,  some  “back-story  ”  describing  the  situation  that  motivates  a  decision,  usually 
containing  some  important  cues,  and  second,  a  description  of  the  decision  required  of  the 
trainee] 

Decision  time  allowed: 

[Usually  set  to  just  enough  time  for  the  trainee  to  queiy  the  assets  (e.g.  unmanned 
vehicles)  needed  to  make  the  decision,  plus  perhaps  one  minute  of  “thinking  time  ”] 

Choices  Available:  (maximum  of  five) 


CHOICE 

NEXT  DECISION 

POINT 

[Brief  phrase  describing  a  choice] 

[#  of  next  decision  point  if 
this  alternative  is  chosen] 

[Brief  phrase  describing  a  choice] 

[Brief  phrase  describing  a  choice] 

Information  Assets  Available,  Routes,  Times,  Reports,  and  Other  Notes: 

[This  section  lists  the  information  assets  that  will  be  made  available  to  the  trainee.  The 
information  that  should  be  specified  for  each  information  asset  is  described  in  the  table 
below] 
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ASSET  TYPE 

INFO  TO  SPECIFY 

SUGV,ARV,  UAV 

Describe  (or  attach  graphic)  and  name  the  possible  routes 
associated  with  this  vehicle. 

For  each  route,  note  whether  any  photographs  or  video  clips 
are  available.  Describe  each  photograph  or  video  clip,  or 
provide  graphic. 

For  each  route,  specify  how  long  the  asset  will  take  to  complete 
it. 

For  each  route,  give  the  text  of  the  report  from  the  “unmanned 
vehicle  operator”  after  the  route  is  complete  -  this  will  appear 
in  the  Text  Message  list  for  the  trainee  to  see. 

ASSET  TYPE 

INFO  TO  SPECIFY 

UGS 

Provide  the  text  of  the  message  that  will  be  displayed  to  the 
trainee  if  he  clicks  on  (queries)  the  UGS 

Recon  Squad 

Provide  the  text  of  the  message  that  will  be  displayed  to  the 
trainee  if  he  clicks  on  (queries)  the  Recon  Squad 

Generic  Message  (e.g. 

Provide  the  text  of  the  message  that  will  be  displayed  to  the 

radio) 

trainee  if  he  clicks  on  (queries)  the  Generic  Message 

Map  Overlay 

Give  the  name  for  the  map  overlay  (e.g.  fire  support)  and 
provide  a  graphic 

Pushed  Message 

Pushed  messages  are  messages  which  are  automatically 
pushed  into  the  text  message  queue  when  the  scenario 
progresses  to  this  decision  point.  Multiple  messages  can  be 
provided  here. 

By  default,  each  message  is  time  stamped  with  the  current 
value  of  the  mission  clock.  However,  you  can  optionally 
specify  that  the  time  stamp  should  be  some  number  of  minutes 
ahead  of  or  behind  the  mission  clock,  e.g.  +45,  -15 

SITREP 

There  is  no  information  that  needs  to  be  provided  for  the 

SITRFP,  other  than  that  one  should  be  an  “asset”  for  this 
decision 

Observer 

Identify  (on  the  map)  the  observation  posts  that  will  be 
available  as  locations  to  which  the  trainee  can  “send”  the 
observer. 

For  each  observation  post,  note  the  message  that  will  be  sent  to 
the  trainee  from  an  observer  at  that  observation  post 
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Asset  Query  Feedback  Rules: 


ALTERNATIVE  CHOSEN 

ASSET  1 

ASSET  2 

ASSET  N 

RESULT 

[First  alternative  from  the 

Choices  Available  table  above] 

Yes/No/ 

Don ’t  Care 

Yes/No/ 

Don 't  Care 

Yes/No/ 

Don ’t  Care 

R1 

[Second  alternative  from  the 
Choices  Available  table  above] 

Yes/No/ 

Don ’t  Care 

Yes/No/ 

Don ’t  Care 

Yes/No/ 

Don 't  Care 

R2 

Hasty  Attack 

Yes 

No 

Don 't  Care 

R3 

In  this  table,  you  can  think  of  the  rows  as  "rules"  consisting  of  conditions  and  actions. 
The  software  starts  at  the  top  and  goes  down  the  list  of  rules,  looking  for  one  whose 
condition  matches  what  the  trainee  actually  did.  The  columns  to  the  left  of  the  Result 
column  comprise  the  condition,  while  the  Result  column  specifies  the  "action",  which  is 
always  to  display  the  corresponding  line  of  text  shown  in  the  Result  table. 

The  "condition  columns"  betM>een  the  Choices  and  Results  columns  always  correspond  to 
sensors,  or  information  assets,  which  can  be  as  simple  as  a  message  from  another  unit. 
Entries  in  these  columns  are  always  one  of:  Yes,  No,  or  Don't  Care.  A  "Yes"  entry 
matches  the  case  where  the  trainee  queried  the  information  asset;  a  "No"  matches  the 
case  where  he  didn't;  and  a  "Don't  Care"  is  a  match  whether  the  trainee  did  or  did  not 
query  the  asset. 

Thus,  taking  the  third  row  in  the  table  above  as  an  example,  if  during  the  course  of  this 
placeholder  scenario,  the  trainee  chooses  Hasty  Attack  from  the  alternatives  presented  to 
him,  and  he  DID  check  Asset  1  and  DID  NOT  check  Asset  2,  the  third  rule  matches  and 
the  feedback  labeled  R3  will  be  displayed.  Note  that  since  Asset  n  is  "Don ’t  ’  Care  ”  in 
this  roM>,  there  is  a  match  regardless  of  what  the  trainee  did  with  Asset  n. 

The  table  should  account  for  every  possible  combination  of  trainee  inputs.  If  the 
trainee ’s  actions  do  not  match  any  row  in  the  table,  no  feedback  will  be  given  to  him. 

The  number  of  rows  required  is: 

(number  of  decision  options)  *  2^  (number  of  asset  columns) 

For  example,  if  there  are  three  decision  alternatives,  and  a  UGS  and  aUAV  with  two 
possible  routes  that  can  be  queried,  the  number  of  rows  in  the  decision  table  should  be  3 
*  2'^3  =  24.  The  use  of  Don ’t  Care  entries  can  substantially  reduce  the  number  of  rows 
in  the  table. 


Asset  Query  Feedback  Text: 


RI 

Text  that  should  be  displayed  to  the  trainee  as  feedback 

R2 

Text  that  should  be  displayed  to  the  trainee  as  feedback 

R3 

Text  that  should  be  displayed  to  the  trainee  as  feedback 

Additional  map  graphics  for  this  decision  node: 

Describe  any  graphics  that  should  be  added  to  the  base  map  for  this  decision  node.  For 
example,  if  the  decision  asks  the  trainee  to  consider  using  a  MEDEVAC  helicopter,  we 
may  want  to  show  its  location  on  the  map.  These  graphics  are  distinct  from  a  “map 
overlay  information  asset”  as  described  above,  in  that  these  graphics  are  always  shown, 
without  the  trainee  having  to  click  on  a  button  to  see  them. 

Trade  Space  Analysis: 

The  trade  space  identifies  three  to  five  tradeoffs  that  the  trainee  should  be  thinking  about 
in  making  this  decision.  We  can  adjust  the  “settings  ”  of  these  factors  in  the  decision 
prompt,  and  the  correctness  of  the  trainee ’s  response  depends  on  the  settings.  For 
example,  for  a  decision  about  whether  to  use  an  organic  Class  lUAVor  ask  for  a 
company  level  Class  II UAV,  the  trade  space  might  include  range  (how  far  is  the  trainee 
from  the  desired  recon  target?  Can  a  Class  I  UAV  go  that  far?),  time  on  station  (can  the 
UAV  loiter  until  the  trainee  gets  there?),  desired field  of  view  (Class  II  will  likely  fly 
higher  and  provide  a  wider  but  less  detailed  view),  and  stealth  (higher  flying  UA  Vs  are 
harder  to  see).  The  correct  answer  depends  on  how  we  tweak  the  particulars  of  the 
factors  in  the  trade  space. 

One  of  the  benefits  of  this  approach  is  that  once  we  have  crafted  a  decision,  we  can 
relatively  easily  create  a  whole  “family”  of  similar  but  distinctly  different  decision  points 
by  varying  the  parameters  in  the  trade  space.  In  this  way,  we  get  significantly  more 
mileage  out  of  the  effort  required  to  set  up  a  decision  point. 

The  trade  space  can  be  described  in  narrative  form,  if  desired,  or  a  table  such  as  this  one 
can  be  used: 


FACTOR 

MY  CLASS  I  UAV 

COMPANY’S  CLASS  II  UAV 

Flight  time 

45  minutes 

3  hours 

Speed 

60  kph 

120  kph 

Stealth 

Audible  and  Visible 

Inaudible,  nearly  invisible 

Elevation/Field  of 
view 

200m/40  degrees  -  can 
resolve  objects  as  small 
as  a  man 

1500m/40  degrees  -  can  resolve  objects 
as  small  as  a  building 

Availability 

Use  anytime 

Must  ask  for  permission,  compete  with 
other  platoons  for  access 
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